{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bFj2ZAU6BtiR"
      },
      "source": [
        "<h1 align=\"center\">\n",
        "  <a href=\"https://portkey.ai\">\n",
        "    <img width=\"300\" src=\"https://analyticsindiamag.com/wp-content/uploads/2023/08/Logo-on-white-background.png\" alt=\"portkey\">\n",
        "  </a>\n",
        "</h1>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5Mfpp2FLAYEA"
      },
      "source": [
        "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1pykKqel2h6ltbVok4nKHhWljcs9_PKDD?usp=sharing)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ekkvItFsWyQL"
      },
      "source": [
        "# Portkey + Nvidia NIM"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "x8RrtT5yYEev"
      },
      "source": [
        "[Portkey](https://app.portkey.ai/) is the Control Panel for AI apps. With it's popular AI Gateway and Observability Suite, hundreds of teams ship reliable, cost-efficient, and fast apps.\n",
        "\n",
        "With Portkey, you can\n",
        "\n",
        " - Connect to 150+ models through a unified API,\n",
        " - View 40+ metrics & logs for all requests,\n",
        " - Enable semantic cache to reduce latency & costs,\n",
        " - Implement automatic retries & fallbacks for failed requests,\n",
        " - Add custom tags to requests for better tracking and analysis and more.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n1kPDnimZIXb"
      },
      "source": [
        "## Quickstart\n",
        "\n",
        "Since Portkey is fully compatible with the OpenAI signature, you can connect to the Portkey AI Gateway through OpenAI Client.\n",
        "\n",
        "- Set the `base_url` as `PORTKEY_GATEWAY_URL`\n",
        "- Add `default_headers` to consume the headers needed by Portkey using the `createHeaders` helper method."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rQSrrUBwkmOP"
      },
      "source": [
        "You will need Portkey and Nvidia API keys to run this notebook.\n",
        "\n",
        "- Sign up for Portkey and generate your API key [here](https://app.portkey.ai/).\n",
        "- Get your Nvidia NIM key [here](https://nvidia.com/dash/api_keys)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "fffx7Tc2ghTR",
        "outputId": "23ed5e5b-f4d6-424a-df17-547913f2ef81"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m405.9/405.9 kB\u001b[0m \u001b[31m5.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m328.5/328.5 kB\u001b[0m \u001b[31m8.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m75.6/75.6 kB\u001b[0m \u001b[31m2.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m12.7/12.7 MB\u001b[0m \u001b[31m28.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m77.9/77.9 kB\u001b[0m \u001b[31m3.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m58.3/58.3 kB\u001b[0m \u001b[31m3.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h"
          ]
        }
      ],
      "source": [
        "!pip install -qU portkey-ai openai"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ptP4L78HlBUL"
      },
      "source": [
        "## With OpenAI Client"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "VZ7zSwDAhc18"
      },
      "outputs": [],
      "source": [
        "from openai import OpenAI\n",
        "from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders\n",
        "from google.colab import userdata\n",
        "\n",
        "client = OpenAI(\n",
        "    api_key= userdata.get('NVIDIA_API_KEY'), ## replace it your Mistral API key\n",
        "    base_url=PORTKEY_GATEWAY_URL,\n",
        "    default_headers=createHeaders(\n",
        "        provider = \"openai\",\n",
        "        custom_host = \"https://integrate.api.nvidia.com/v1\",\n",
        "        api_key= userdata.get('PORTKEY_API_KEY'), ## replace it your Portkey API key\n",
        "    )\n",
        ")\n",
        "\n",
        "completion = client.chat.completions.create(\n",
        "  model=\"google/gemma-2-27b-it\",\n",
        "  messages=[{\"role\":\"user\",\"content\":\"Write a limerick about the wonders of GPU computing.\"}],\n",
        ")\n",
        "\n",
        "print(completion.choices[0].message.content)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wMhOOkaLqkSp"
      },
      "source": [
        "## Observability with Portkey"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LzECvXQaqr6Z"
      },
      "source": [
        "By routing requests through Portkey you can track a number of metrics like - tokens used, latency, cost, etc.\n",
        "\n",
        "Here's a screenshot of the dashboard you get with Portkey!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W80YWspqr7s0"
      },
      "source": [
        "![Screenshot 2024-04-10 at 4.32.34 PM.png]()\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "yqrIH5mY01BI"
      },
      "outputs": [],
      "source": []
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
